A Task-Driven Framework for Driving Simulation: Scenario Orchestration with Autonomous simulated Vehicles (SOAV)

1 Scenarios in driving simulators cover what the human participants experience and what the re2 searchers need: the physical scene, pre-defined traffic flow, simulated vehicles’ interactions with 3 the participants and measurements to be collected. 4 Previous methodologies for orchestrating scenarios regarding the interactions have the fol5 lowing drawbacks: 1) action sequences that simulated vehicles should follow lack the contexts of 6 each action; 2) programming languages always include platform-dependent details and are not suit7 able for context modelling and 3) scenarios cannot be generated dynamically to cope with failures 8 that happen in trials. 9 To overcome the limitations above, an Ontology for Scenario Orchestration (OSO) was 10 first developed to model concepts and their relationships in the domain of scenario orchestration, 11 including a concept named Assignment, which represents the task(s) of virtual drivers and en12 codes the contextual information of proposed actions, e.g., simulated vehicles involved. It can also 13 provide a file for machine processing. 14 An algorithm named NAUSEA (autoNomous locAl manoeUvre and Scenario orchEstration 15 based on automated action plAnning) was generated to utilise Assignments recorded using OSO. 16 Encoded in the driver model SAIL (Scenario-Aware drIver modeL), NAUSEA can be used by a 17 virtual driver to control simulated vehicles dynamically. Failed Assignments, designed to generate 18 specific interactions, can be re-tried if permitted. A framework SOAV (Scenario Orchestration with 19 Autonomous simulated Vehicles) was formed to support SAIL/NAUSEA and orchestrate scenarios 20 with autonomous vehicles. 21 Three verification experiments showed that SOAV worked properly by generating desired 22 interactions and dealing with failures. OSO can also provide contextual information in a human23 readable and machine processable manner. 24 TRB 2014 Annual Meeting Paper revised from original submittal. Xiong, Z., Carsten, O., Jamson, H., Cohn, A.G. 2 INTRODUCTION 1 By imitating driving activities of the real world, driving simulators can: 2 • have reproducible and consistent scenarios; 3 • have a substantially risk-free environment; 4 • provide some hazardous situations not easily performed in the real world. 5 A scenario is then the key to provide a pre-defined environment that experimenters need 6 a participant to experience. It includes the physical scene, pre-defined traffic flow, simulated ve7 hicles’ interactions with the participant’s vehicle1 and measurements that need to be collected. 8 Choreography of scenarios, therefore, plays an important role in driving simulation. In general, 9 the interactions in scenarios should be reproducible and the human participants should be able 10 to make similar decisions in driving simulators as they would in the real world, so there are two 11 requirements when it comes to scenarios in driving simulation: realism and reproducibility. 12 Realism indicates the requirement of adopting sufficient driving behaviours. This term is 13 motivated by the work from (1), which states that “In our experience with research studies in high14 fidelity simulators, users generally focus their evaluation of the model realism towards the richness 15 of the behaviors, not their fidelity.” 16 On the other hand, reproducibility means that the essential conditions in a scenario should 17 be repeatable in each trial (2). However, the balance between realism and reproducibility can be 18 hard to define because when scenarios are running, unexpected movement of vehicles, including 19 the participant’s, may interrupt some pre-defined interactions. As a result, behaviours of each 20 simulated vehicle are always limited in order to guarantee reproducibility. 21 This research, therefore, is trying to find a way of combining realism with reproducibility 22 by orchestrating scenarios with autonomous simulated vehicles around the participant’s vehicle. 23 By using an algorithm, it is possible to engage the participant’s vehicle actively based on rich 24 behaviours and pre-defined tasks in order to create desired situations for interactions. During the 25 process of developing the algorithm, a framework for scenario orchestration was developed. 26 This paper presents some major concepts and components in SOAV. “Related Work” Sec27 tion introduces some background and is followed by a description of the framework SOAV (Sce28 nario Orchestration with Autonomous simulated Vehicles). Verification and results will be pre29 sented in “Verification and Results” Section. Conclusion and future research will be discussed at 30 the end. 31 RELATED WORK 32 In driving simulation, orchestration of scenarios has been a focus of research since the mid 1990s 33 regarding how scenarios can be described and how to use those descriptions (e.g., (3)). Less effort 34 has been put into this area in recent years since existing methodologies seem adequate enough for 35 applications. Generally speaking, these methodologies share the same idea: have humans describe 36 every aspect of the scenario, and then author the scenario to relevant simulated vehicles. This 37 process can define the rules or sequence of actions that the simulated vehicles should follow before 38 or when the scenario is exposed to human participants. 39 In general, three questions need to be answered in order to orchestrate a particular scenario: 40 1In this research, only one participant or participant’s vehicle (simulator) is included in a particular scenario. TRB 2014 Annual Meeting Paper revised from original submittal. Xiong, Z., Carsten, O., Jamson, H., Cohn, A.G. 3 • which simulated vehicles will be involved in the scenario? 1 • how will those simulated vehicles be prepared? 2 • how can the scenario be produced? 3 Most researchers put a focus on answering individual question with some specific algo4 rithms as listed below. 5 Question one can be termed “actor management problem”. Simulated vehicles that will 6 interact with human participant’s vehicle can be pre-defined and fully described beforehand (e.g., 7 (4)). Alternatively, an actor management algorithm was developed in (5) by considering the aver8 age speeds of simulated vehicles needed to reach the proposed location for interactions. A simu9 lated vehicle with an inconspicuous speed trajectory, which is not noticeable to human participants, 10 can be assigned to interact with the participant. 11 Question two can be termed “actor preparation problem” and is related to the “realism” part 12 of the scenario, as in this phase simulated vehicles can have some autonomy to prepare themselves. 13 In (5), some work was done regarding how to prepare simulated vehicles in an inconspicuous 14 manner by adopting a specific speed trajectory towards the required location for interactions. 15 Question three can be termed “scenario execution problem” and is related to three aspects: 16 what are available actions, how to trigger an action and how to schedule those actions. In pre17 vious work (e.g., (6)), the event-driven mechanism was widely used and can be regarded as pre18 conditions for particular actions: conditions must be true before these actions are applied. More19 over, the scheduling of such actions is often crafted by humans in order to specify the execution 20 order of actions or sub-scenarios (e.g., (7)). 21 None of these questions have covered scenario failures and some other scenario develop22 ment issues as discussed in (8), such as the following examples: 23 • Pre-Run Issues: 24 – the experimenter describes interaction outcomes without corresponding context and, 25 – the experimenter fails to correctly predict human participants’ reactions. 26 • Run-Time Issues: 27 – participants do not want to be engaged in some interactions and, 28 – some interactions never happen due to design or system issues. 29 Pre-run issues suggest that an efficient communication mechanism is lacking. Experi30 menters may fail to foresee the extra actions from a participant as they focus on outcomes only; the 31 experimenters do not have a general picture of the capacities of the scenario orchestration mecha32 nism and the simulation software. This communication problem can make the scenario orchestra33 tion process time-consuming and make the scenario liable to run-time issues, so there needs to be 34 some ways to share knowledge such that both experimenters and programmers can have some pre35 knowledge, which includes, but is not limited to, the capacities of the driving simulator software, 36 the potential pre-conditions and the effects of particular interactions. 37 Run-time issues invariably cause failures, e.g., a trigger does not fire or participants do not 38 want to be engaged. Those failures are hard to avoid, but there is a possibility that they can be 39 TRB 2014 Annual Meeting Paper revised from original submittal. Xiong, Z., Carsten, O., Jamson, H., Cohn, A.G. 4 fixed dynamically without help from humans by automatically re-orchestrating the scenarios, if 1 permitted. 2 In order to deal with the four issues above, both run-time and pre-run ones, a framework 3 SOAV (Scenario Orchestration with Autonomous simulated Vehicles) was developed and its two 4 major components: the Ontology for Scenario Orchestration (OSO) and an algorithm named NAU5 SEA (autoNomous locAl manoeUvre and Scenario orchEstration based on automated action plAn6 ning) encoded in a driver model named SAIL (Scenario-Aware drIver modeL) have been elaborated 7 in (9, 10, 11) . 8 OSO is used to describe scenarios and relevant driving context to a virtual driver in a formal, 9 context-oriented, programming-independent, logic-based, human-understandable and machine10 processable manner. SAIL/NAUSEA is utilised by virtual drivers to carry out Assignments, which 11 are tasks that virtual drivers need to accomplish in order to generate the required interactions with 12 the human participant. Assignments can provide relevant contextual information to virtual drivers: 13 the Formation Position, Monitor(s), Success Condition(s), Failure Condition(s), Assignment-action(s) 14 and the measurement from the interaction generated by this Assignment. 15 In general, SOAV can provide the following features: 1) the simulated vehicles that are 16 controlled by virtual drivers can actively engage the participant to avoid failure and 2) if failures 17 happen, the simulated vehicles can be re-driven to generate the proposed interactions by virtual 18 drivers. However, SOAV still needs further development. In the rest of this paper, the framework 19 of SOAV will be described with more details compared to (9, 10, 11). All three experiments that 20 have been carried out to verify SOAV will be introduced and future research will also be discussed 21 at the end. In the next section, some domain concepts will be introduced first, which have been 22 encoded in OSO and formed the basis for this research. 23 FRAMEWORK DESCRIPTION 24 Generally speaking, SOAV is a framework used to orchestrate scenarios with autonomous simu25 lated vehicles in driving simulation. It was formed in order to support and test a decision making 26 algorithm, which needed a driver model to provide it with data for decision making and module for 27 decision execution. This driver model was used to develop a Virtual Driver. Some other facilities 28 were also implemented in order to control the simulated vehicles in the simulation according to the 29 decisions sent from the Virtual Driver. 30 Main Concepts 31 Virtual Driver 32 A Virtual Driver in this research indicates an intelligent controller that can be used to make driving 33 decisions or carry out pre-scheduled actions based on scenario requirements. 34 Driving Decisions indicate actions that should be undertaken to drive safely and satisfy 35 scenario requirements. The actions that this research will use are: overtaking, speed adap36 tation and lane-changing; 37 Pre-scheduled Actions indicate some actions required to change the states of objects in 38 driving simulation in order to generate specific interactions, e.g., request to “set the desired 39 speed of the simulated vehicle no.1 as 30 mph” or request to “place cones in a specific 40 road segment whose id is ‘r3.2’ ”; 41 TRB 2014 Annual Meeting Paper revised from original submittal. Xiong, Z., Carsten, O., Jamson, H., Cohn, A.G. 5 That is, this Virtual Driver should be able to interfere with the simulated vehicles’ au1 tonomous decision-making process and change the states of some objects in driving simulation, 2 rather than the simulated vehicles themselves. 3 Monitor System 4 Monitors are used to supervise the state variables in simulation, e.g., vehicles’ speeds and positions. 5 A Monitor System can be developed based on the work from (12), which identified four kinds of 6 Monitors: 7 When The Monitor will be activated once if its condition is satisfied. The condition is a 8 state transition. 9 Every The Monitor will be activated repeatedly if its condition is satisfied. The condition 10 is a state transition. 11 Aslongas The Monitor will be activated during the time when its condition is satisfied; 12 Whenever The Monitor will be activated repeatedly if its conditions is satisfied. 13 As a result, the Monitors can supervise state transitions happening at any Instant or state 14 status during any time intervals. The four keywords representing the four types of Monitors will 15 be called Monitor operators in this research. 16 Flock and Ego-Vehicle/Flock 17 A Flock refers to a platoon of simulated vehicles. Its leader’s states, e.g., position, speed, etc., will 18 be used to represent the Flock’s states in this research. A scenario can have several Flocks. 19 The simulated vehicle or Flock the Virtual Driver is controlling is termed the “Ego-vehicle” 20 or the “Ego-flock” respectively. The leader of the Ego-flock is driven by a Virtual Driver, who will 21 manage that Flock with appropriate orders. Other members of the Ego-flock will simply adopt the 22 same orders from the Virtual Driver. 23 Action 24 An Action is what a Virtual Driver can do to change the state of the simulated world. Actions are 25 divided into two categories: High-Level and Low-Level. 26 A Low-Level Action is defined as an Action that can only be performed in one way, one 27 sequence and by one entity. Examples of Low-Level Actions are: “Set Steering Angle”, “Set 28 Speed”, “Set Acceleration Rate”, “Set Action Lane”, “Create an Object” (e.g., a vehicle or a road 29 segment), “Set Action Status”, etc. 30 High-Level Actions need Recipes (13), each of which is a set of Low-Level Actions that 31 specify how to perform a High-Level Action. For instance, “Block” is a High-Level Action that can 32 be designed as: three simulated vehicles pass the participant’s vehicle to prevent it from overtaking 33 its leader vehicle. 34 Each Action contains some information that can be used by a Virtual Driver: 1) the name/id 35 of the Ego-vehicle/flock; 2) the proposed deadline of the Action; 3) the proposed Duration of the 36 Action; and 4) the proposed release time of the Action. Moreover, each Action has a type and an 37 Action profile, e.g., “change desired speed to 10 mph” is an Action whose type is “Low-Level” and 38 “Adapt-Speed”. It has an Action profile of setting “Desired Speed” with a value of “10.0” (mph). 39 TRB 2014 Annual Meeting Paper revised from original submittal. Xiong, Z., Carsten, O., Jamson, H., Cohn, A.G. 6 Trigger 1 Monitors mentioned in Section “Monitor System” can be used to establish Triggers: Pre-conditions, 2 Success Conditions and Failure Conditions. 3 Because of the importance of Pre-conditions that can make a Virtual Driver perform par4 ticular Actions when some conditions are satisfied, the word Monitor has been used to specifically 5 refer to Pre-conditions. Success Conditions and Failure Conditions, on the other hand, specify 6 Triggers that can indicate whether or not an Action has succeeded and failed respectively. All 7 three kinds of Triggers should include the following information: 8 • if the Trigger should dictate a state or an event (state transition). 9 • which object the Virtual Driver should supervise, e.g., a simulated vehicle or a Flock; 10 • which state variable of the object should be dictated and compared to some threshold 11 value, e.g., speed, position; 12 • what threshold value should be compared to and, 13 • how to dictate the identified state variable: when, whenever, aslongas, every, see Figure 14 ?? on Page ?? for more details. 15 As there are some common events in simulation, e.g., a vehicle enters or leaves a zone, 16 Triggers can also include some Trigger type, which can be: “Cross (a pre-defined line)”, “Enter (a 17 pre-defined zone)”, “Exit (a pre-defined zone)” or “Timer” (12). 18 In addition, this research has put a focus on dealing with failures caused by triggers that 19 do not fire and participants that do not want to be engaged, so failure conditions represent some 20 special states of objects in The Sim when an Assignment is not fired. 21 Formation Position 22 Inspired by (14), the concept of Formation Position (Figure 1) has been proposed to specify the 23 spatial goal of a Virtual Driver. For instance, if the Virtual Driver’s spatial goal is “Follower”, then 24 he/she will try to drive the Ego-vehicle/flock to the “Follower” position as specified in Figure 1. 25 Because Formation Position is a set of pre-defined relative local positions around the simulator 26 driver/participant’s vehicle, no assumption has been made regarding the absolute spatial relation27 ships between them. However, according to specific applications, values can be assigned in order 28 to indicate how far those positions should be. For instance, 200 metres can be used to indicate the 29 optimal distance between the “Leader” position to the Participant’s vehicle’s position. 30 Role Matching 31 This is a process used to decide which simulated vehicle a Virtual Driver should control. It is 32 determined by required vehicle type, Formation Position and the required speed. A successful 33 Role Matching should identify any simulated vehicles that 1) are the required vehicle type, e.g., 34 Volvo S40; 2) are at or near the required Formation Position and 3) have sufficient maximum speed 35 that can reach the required Formation Position in time. A description of this process can be found 36 in (9). 37 TRB 2014 Annual Meeting Paper revised from original submittal. Xiong, Z., Carsten, O., Jamson, H., Cohn, A.G. 7 FIGURE 1 Formation Position Assignment 1 An Assignment is the task that a Virtual Driver needs to carry out. Every scenario contains at 2 least one Assignment. An Assignment includes three kinds of information: general information, 3 Triggers and Assignment-actions. 4 Firstly, a Virtual Driver should know the following facts: 5 • the state of an Assignment, which can be: 6 Initial indicates that the Triggers in the Assignment should not be monitored at present; 7 Pending indicates that the Triggers in the Assignment are being dictated or should be 8 dictated from the next decision loop; 9 Failure indicates that this Assignment is failed because of the failure of its Assignment10 action(s) or the Failure Conditions are true; 11 Success indicates that this Assignment is now finished and has succeeded; 12 • any requirements regarding the Ego-vehicle/flock, e.g., required vehicle type; 13 • the spatial goal of the Virtual Driver where the Ego-vehicle/flock should be driven to 14 before the execution of some Assignment. This spatial goal is based on the Formation 15 Position in Figure 1; 16 • how many times the Virtual Driver can try to finish this Assignment; 17 • the pre-defined Ego-vehicle/flock for the Assignment. If it is not specified, the Virtual 18 Driver will look for candidates dynamically. 19 Secondly, a Virtual Driver should know any Triggers that include 1) the Monitor that is 20 used to activate Assignment-actions and 2) the Success and Failure Conditions that indicate the 21 accomplishment or abortion of its parent Assignment. 22 TRB 2014 Annual Meeting Paper revised from original submittal. Xiong, Z., Carsten, O., Jamson, H., Cohn, A.G. 8 Finally, a Virtual Driver should know what Actions to undertake in an Assignment, these 1 are detailed in the Assignment-actions. Assignment-actions can be driving actions, e.g., change2 lane, or non-driving actions, e.g., “create new vehicle”, “collect driver’s reaction time (braking)”, 3 “create a road segment”, “activate traffic flow one”, “activate Assignment two” or “activate Situa4 tion one”. Assignment-actions can be triggered and executed once unless a failure has been iden5 tified. Repetition of Assignment-actions is possible by adopting an indicator stating how many 6 times the Assignment-action can be repeated, but in this research, this is ignored. 7 Assignments can be used not only by a Virtual Driver to generate any required interactions 8 with the capability of re-generating those interactions, but also by experimenters to describe inter9 action context along with outcomes (measurements or Actions needed). They can also predict the 10 reactions from participants by providing Success/Failure Conditions, so the pre-run issues can be 11 dealt with by utilising Assignments to encode and notify 1) the interaction outcomes along with 12 corresponding context and 2) the potential reactions from participants. 13 General Plan 14 A Virtual Driver should have some plan to guide its behaviours so a concept of the General Plan 15 was introduced. It is a temporal constraint graph Grα ( see (13) and (9) for more details) and is built 16 from several pre-defined Actions and Assignments. It contains metric and precedence constraints. 17 For instance, “Action α before Action β” is a precedence constraint; “ start time of β finish time 18 of α 6 100 (seconds)” is a metric constraint. There are two types of metric constraints: duration 19 and delay. Delay is the period between two Instants that belong to different time intervals, while 20 duration is the period between two Instants that are within the same time interval. 21 In every scenario, each Virtual Driver needs to finish a top High-Level Action α that can 22 be either Perform-scenario or Free. Perform-scenario has only one Recipe that contains four sub23 Actions, namely, β0, β1, β2 and β3 as shown in Figure 2. Free makes the Virtual Driver ignore any 24 Assignments and autonomously navigate the world, in which case the route or the destination will 25 be based on a pre-defined route. Recipes are not needed for Free. 26 FIGURE 2 Action Recipe for the Virutal Driver (Perform-sceanrio) β0 (Get-to-the-initial-state) adopts the initial state (e.g., initial speed, initial target speed 27 etc.). β1 (Generate-formation) means that a Virtual Driver should navigate the Ego-vehicle/flock 28 to the proposed Formation Position in order to perform the corresponding Assignment-actions. 29 Because the recipe of β1 will change according to the dynamic environment, this Action will 30 not be further divided into sub-Actions, but will be monitored throughout the scenario in order 31 to make sure that the Ego-vehicle/flock can get to the Formation Position in time. β2 means 32 Perform-assignment Action, and can be further divided into several Assignment-actions, which 33 TRB 2014 Annual Meeting Paper revised from original submittal. Xiong, Z., Carsten, O., Jamson, H., Cohn, A.G. 9 are represented as γ0 through γn (n is the number of Assignment-actions a Virtual Driver needs to 1 carry out). Each Assignment-action (γ0 through γn) is contained in a corresponding Assignment. β3 2 (Clean-up) should be specified by experimenters as an Assignment-action in most circumstances; 3 however, it can be an autonomous Action by changing it to the top-Action of Free. In this research, 4 some High-Level Actions in the Recipe of α will not be further refined, i.e., β0, β1 and β3, as they 5 change according to dynamic environment. Moreover, all Assignment-actions have been regarded 6 as Low-Level Actions in his research. 7 Scenario 8 A scenario is a pre-defined environment and situation(s) that experimenters need a participant to 9 experience. It includes the physical scene, pre-defined traffic flow, simulated vehicles’ interactions 10 with the only one participant’s vehicle and measurements that need to be collected. 11 The interactions in a scenario are generated by the Virtual Driver, who changes the state 12 of the environment by controlling simulated vehicles or modifying the physical scene, e.g., place 13 cones on the road. In the former case, the Virtual Driver controls the Ego-vehicle/flock to produce 14 interactions with the participant’s vehicle and in the latter case, the Virtual Driver will request 15 corresponding facilities to modify the physical scene. 16 A scenario can have several Assignments performed by the Virtual Driver, so a scenario 17 can have different interactions, aiming at providing individual measurements in one trial or run. 18 For instance, a particular scenario can have two interactions, the participant should first be blocked 19 by a lorry at some position for 10 seconds, after which, the participant should experience a five 20 minutes’ free drive. The second interaction should happen right after the free drive: a lead car 21 should break down and force the participant to brake. 22 In the following section, components of SOAV will be introduced. 23 Components of SOAV 24 SOAV is generally divided into two parts: Offline and Online (Figure 3). Descriptions of each 25 component are as follows: 26 Offline Component 27 The Offline Component is in charge of describing scenarios using OSO (Ontology for Scenario 28 Orchestration) based on Protégé (15), which outputs an SDF (Scenario Definition File) with the 29 RDF/OWL syntax recorded in an XML file2. As how OSO is recorded does not affect the work 30 in this research, RDF, OWL and XML will not be introduced. Details of those languages can be 31 found in (16). 32 Protégé is an ontology editor and knowledge acquisition system developed by Stanford 33 University and the University of Manchester. It is free, Open Source, widely used and well sup34 ported, so Protégé has been chosen as the editor for OSO and thus the scenario orchestration tool. 35 Moreover, Protégé also provides reasoners, e.g, HermiT3, to check a particular ontology’s sub36 sumption: if one class can be a subclass of another class or its consistency: if there are any classes 37 that cannot have any individuals. 38 2RDF is short for Resource Description Framework; OWL stands for Web Ontology Language; XML stands for Extensible Markup Language. 3http://protegewiki.stanford.edu/wiki/HermiT TRB 2014 Annual Meeting Paper revised from original submittal. Xiong, Z., Carsten, O., Jamson, H., Cohn, A.G. 10